1. Field of the Invention
The present invention relates generally to data computing environments and, more particularly, to a system providing methodology for moving a project in a complex event processing cluster.
2. Background Art
Historically, most businesses were most concerned about the need for accurate storage, retrieval, and analysis of data. Today, businesses are operating in environments where the need to monitor events comes at an ever-accelerating pace.
Event-driven systems have typically been built around either relational databases or real-time messaging systems, or a combination of both. While these technologies have their advantages, neither is particularly well suited for managing and analyzing events in rapidly changing environments. Relational database servers can process large amounts of stored data and can analyze the information with relative ease but are not designed to operate in real-time environments, and do not provide an effective way to monitor rapidly changing data. Messaging systems permit data to be monitored in real time but generally are not capable of complex computations, correlations, pattern matching or references to historical data.
Custom applications must often be combined with these technologies to create a viable solution. However, the use of custom applications to compensate for the limitations of these technologies creates new complications. For example, custom applications become increasingly complex very quickly as an organization's need for progressively more sophisticated analysis grows. Further, custom applications are also costly to modify, and they do not scale well as organizational needs change.
Tools, such as the Sybase CEP Engine available from Sybase, Inc., of Dublin, Calif., have been developed in the field of Complex Event Processing (CEP) and Event Stream Processing (ESP) applications for use in building applications to process and analyze large volumes of fast-moving data in highly dynamic environments. These tools couple the best features of relational databases and messaging technologies, combined with many new features required for the rapidly changing world faced by today's businesses. For example, like a real-time messaging system, reactions to each new piece of data occur immediately. Like a relational database system, support exists for the use of a query language similar to SQL to process and analyze incoming data, and to correlate it with historical data. Also, sophisticated pattern-matching capabilities are provided without requiring large amounts of custom application code.
FIG. 1 illustrates a block diagram of components of an end-to-end application incorporating a complex event processing engine 100. Messages travel through the application starting from an external data source 110 to an input adapter 120. The input adapter 120 may poll the external source 110 or register for notifications or use some other mechanism to receive data. The input adapter 120 publishes messages to one or more data streams 130. The stream(s) 130 feed projects and query module(s) 140. The query module(s) 140 publish to one or more streams 150. The stream(s) 150 feed an output adapter 160. The output adapter 160 subscribes to the stream(s) 150, processes the messages, converts the data to a format suitable for the destination 170, and then transmits the output data to the destination 170. The destination 170 performs whatever actions with the data that it is designed to do.
Input and output adapters enable the CEP engine 100 to send and receive messages from dynamic and static external sources and destinations. External sources or destinations can include data feeds, sensor devices, messaging systems, radio frequency identification (RFID) readers, e-mail servers, relational databases, and the like. By way of example, a series of input adapters 120 can translate messages from data sources 110, such as a temperature sensor, bar code scanner and a Java Message Service (JMS) “cloud”, into formats compatible with the CEP engine 100 for processing by several CEP queries 140 with the output adapters 160 converting the CEP result rows into updates to destinations 170, such as a database server, e-mail messages, and data for a web services dashboard.
The CEP engine 100 may process hundreds of thousands of messages per second, with latency measured in milliseconds and may run on a single processor, or may consist of multiple servers (i.e., a server cluster) distributed across multiple machines, each containing one or more processors. When a cluster is utilized, a user (or system) may need to control movement of a running CEP project from one node to another in a cluster, such as to balance the load. One basic way to move a project from one node to another would be to stop the running project in the source node and start the project in the destination node. Unfortunately, in this approach, messages which are currently in transition in the CEP may be lost. One way to avoid such losses is to enable persistence in a CEP project, so that the transit stream messages as well as status of the query engine are persisted, such as to persistent storage 180. However, this will not provide a complete solution, since messages could be in transition in the input 120 and output 160 adapters, and stopping a project may result in those messages being lost.
By enabling guarantee delivery, which is a communications protocol that provides acknowledgements from the recipient to the sender assuring that messages were delivered, message loss could be avoided, so that, together with persistence enabled, a project could be stopped and moved a project successfully. However, once you add persistence and guarantee delivery, system performance decreases significantly. Performance sensitive CEP projects (e.g., those that avoid the use of disk input/output) may not want to add persistence and guarantee delivery, and some projects may enable persistence and not guarantee delivery.
Accordingly, a need exists for an approach to provide efficient and successful project movement in a CEP cluster that avoids the problems of the prior art. The present invention addresses such as need.